Creating a Feasible Corpus for Persian POS Tagging

This paper describes creation of a test collection for Persian Part of Speech Tagging experiments. This collection was created by modifying a manually Part of Speech (POS) tagged Persian corpus with over two million tagged words. The original collection had a tag set of 550 tags that are more than what any machine learning algorithm can handle. The number of tags for these experiments was reduced to only 40 tags. Since the main motivation for creation of this collection is building and testing efficient Persian Part of Speech taggers, a benchmark tagger was created for that used Maximum Likelihood Estimation (MLE) for guessing the correct tags in Persian. Two different versions this tagger were produced that differed on handling “Unseen words”. One of them tagged the unseen words with a “DEFAULT” tag and in the other one with “N-SING” which was the most frequent tag used in the corpus. The best accuracy that was achieved by MLE tagging was (95.43%). The results of the experiments are encouraging as they are comparable to other languages such as English German and Spanish. This test corpus will be distributed among other researchers to be used as a base for Persian natural language processing.